Image output device, method, and recording medium therefor

ABSTRACT

An image output device comprising: a representative parallax acquisition unit; a provisional output parallax determination unit to determine a provisional output parallax for each of the stereoscopic image frames depending on an output condition of the stereoscopic video, based on the representative parallax for each of the stereoscopic image frames; an output parallax adjustment unit; and an output unit, wherein the provisional output parallax determination unit determines the provisional output parallax for a reference frame corresponding to the representative parallax for the reference frame, and determines the provisional output parallax for a target frame corresponding to the representative parallax for the target frame, in accordance with the provisional output parallax defined for each range of the representative parallax for the stereoscopic image frame, the reference frame being sequentially determined from the stereoscopic image frames, and the target frame being the stereoscopic image frame immediately after the reference frame.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2012/079069 filed on Nov. 9, 2012, which claims priority under 35U.S.C §119(a) to Japanese Patent Application 2011-277353 filed on Dec.19, 2011. Each of the above application(s) is hereby expresslyincorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image output, and particularly to anadjustment of a binocular parallax for each stereoscopic image frame ofa stereoscopic video.

2. Description of the Related Art

A stereoscopic image processing device in Japanese Patent ApplicationLaid-Open No. 2004-221699 (hereinafter referred to as PTL 1), when adisplayed subject reaches a limit parallax, generates parallax images inaccordance with acquired adequate parallax information, such that theadequate parallax is implemented in subsequent stereoscopic display. Thecontrol of the parallax is implemented by going back tothree-dimensional data and optimally setting a camera parameter. Here, atwo-dimensional image generation unit of the stereoscopic imageprocessing device calculates a depth Fxy that meets the adequateparallax. When the range of the depth is K1 to K2 and the depth value ofeach pixel is Gxy, Fxy=J1+(Gxy−K1)×(J2−J1)/(K2−K1) holds. In the casewhere Fxy is not an integer, a rounding or a process for decreasing anapproximation parallax is performed.

A three-dimensional image processing device in Japanese PatentApplication Laid-Open No. 6-028452 (hereinafter referred to as PTL 2)provides an observation viewpoint sensor to detect an observationviewpoint of a facing observer, on a display device to display athree-dimensional image shown in perspective by a viewpoint coordinatesystem, and a computer moves the viewpoint coordinate system of thedisplay image, in tune with the observation viewpoint detected by theobservation viewpoint sensor. Thereby, the perspective viewpoint of thethree-dimensional image to be displayed on the display device is movedso as to follow the observation viewpoint of the observer, and the imagedisplay is performed such that the perspective viewpoint always accordswith the observation viewpoint.

A three-dimensional image display device in Japanese Patent ApplicationLaid-Open No. 9-238369 (hereinafter referred to as PTL 3) includes aviewpoint detection device to detect a viewpoint position of anobserver, a picture generation device to generate two pictures having aparallax for the left and right eyes, a picture display device todisplay the two pictures for individually projecting them, and a pictureprojection device to individually project the two displayed pictures tothe left and right eyes of the observer. The picture generation devicegenerates pictures that reflect the change in an observation directionfor an observation object corresponding to the movement of the viewpointof the observer, based on output signals of the viewpoint detectiondevice. Furthermore, the picture generation device generates pictures ofan imaginary body that reflect the change in the observation directionfor the observation object corresponding to the movement of theviewpoint of the observer, and that have a parallax for the left andright eyes of the observer, based on output signals of the viewpointdetection device.

A stereoscopic image display method in Japanese Patent ApplicationLaid-Open No. 8-327948 (hereinafter referred to as PTL 4), by aconfiguration that includes a display unit to provide a liquid crystalshutter in front of a display device and dispose a lenticular lens infront of the liquid crystal shutter, and a control unit to which aviewpoint position of an observer is input, displays a parallax imagecorresponding to the right eye and left eye on the display device bytime division, and synchronously with the parallax image, changes thewidth and position of a light-transmitting part of the liquid crystalshutter in response to the viewpoint position of the observer, allowingthe parallax image to be observed by a corresponding eye through thelenticular lens.

SUMMARY OF THE INVENTION

It may occur that a stereoscopic video using a parallax induces fatigueof a viewer if it is not displayed by an appropriate parallax amount.The appropriate parallax amount varies depending on the size of adisplay device for displaying, the stereoscopic fusion limit of aviewer, and the like. Therefore, it is necessary to perform a parallaxadjustment corresponding to them.

In PTL 1, a depth Fxy meeting an adequate parallax is calculated androunded off. Therefore, it may occur that the parallaxes between framesbecome equivalent and a change in stereoscopic effect involved in aframe transition cannot be sensed, or conversely, that the parallaxeschange too much between frames, leading to fatigue of a viewer. Forexample, when a transition of the parallaxes at the time of image takingas shown in the (a) portion of FIG. 14 is adjusted to display parallaxesas shown in the (b) portion of FIG. 14, in some cases, the parallaxesbetween contiguous frames become equivalent so that the stereoscopiceffect is lost, or conversely, a large variation in the parallaxesoccurs between contiguous frames so that a viewer is fatigued.

The present invention has an object to reflect a transition ofrepresentative parallaxes for a stereoscopic video, in the parallaxadjustment depending on the output condition of the stereoscopic video.

The present invention provides an image output device including: arepresentative parallax acquisition unit to acquire a representativeparallax for each of multiple stereoscopic image frames constituting astereoscopic video; a provisional output parallax determination unit todetermine a provisional output parallax for each of the stereoscopicimage frames depending on an output condition of the stereoscopic video,based on the representative parallax for each of the stereoscopic imageframes acquired by the representative parallax acquisition unit; anoutput parallax adjustment unit to adjust an output parallax for each ofthe stereoscopic image frames, based on the provisional output parallaxfor each of the stereoscopic image frames determined by the provisionaloutput parallax determination unit; and an output unit to sequentiallyoutput the stereoscopic image frame whose output parallax is adjusted bythe output parallax adjustment unit, in which the provisional outputparallax determination unit determines the provisional output parallaxfor a reference frame based on the representative parallax for thereference frame, and determines the provisional output parallax for atarget frame based on the representative parallax for the a targetframe, the reference frame being sequentially determined from thestereoscopic image frames, and the target frame being the stereoscopicimage frame immediately after the reference frame; the output parallaxadjustment unit adjusts a difference between the output parallax for thereference frame and the output parallax for the target frame, based on adifference between the representative parallax for the reference frameand the representative parallax for the target frame; the representativeparallax for each of the stereoscopic image frames includes astatistical operation value to be calculated based on parallaxes that,of parallaxes for the stereoscopic image frame, meet a predeterminedcondition; and the statistical operation value to be calculated based onthe parallaxes that meet the predetermined condition excludes an averageparallax in a predetermined area of the stereoscopic image frame.

Preferably, the statistical operation value includes at least one of amaximum value, a minimum value, a mode value and a median value of theparallaxes for the stereoscopic image frame.

Preferably, the representative parallax for each stereoscopic imageframe includes at least one of a maximum value, a minimum value, a modevalue and a median value of parallaxes for a subject present on abackground side or parallaxes for a subject present on a foregroundside, among the parallaxes for the stereoscopic image frame, thebackground side being a side in a direction farther from an imagingdevice than a cross point, and the foreground side being a side in adirection closer to the imaging device than the cross point.

Preferably, the parallaxes that meet the predetermined condition shouldinclude a parallax for a gazing position to the stereoscopic imageframe.

Preferably, the gazing position includes a gazing point of a viewer ofthe stereoscopic image frame, a gazing point of a videographer of thestereoscopic image frame, or an arbitrary designated gazing point in thestereoscopic image frame.

Preferably, the parallaxes that meet the predetermined conditionincludes parallaxes for a face area, parallaxes for a focusingevaluation value calculation area, parallaxes for a central image area,parallaxes for a subject present on a background side among theparallaxes for the stereoscopic image frame, or parallaxes for a subjectpresent on a foreground side among the parallaxes for the stereoscopicimage frame. The background side is a side in a direction farther froman imaging device than a cross point, and the foreground side is a sidein a direction closer to the imaging device than the cross point.

Preferably, the image output device includes an output-allowableparallax width acquisition unit to acquire an upper limit and a lowerlimit defining an output parallax width, as an output condition of thestereoscopic video. The output parallax width is a width of the outputparallax that is allowable.

Preferably, the image output device includes a parallax width adjustmentunit to adjust a parallax width to the output-allowable parallax widthacquired by the output-allowable parallax width acquisition unit, whenthe parallax width does not comply with the output-allowable parallaxwidth. The parallax width is defined by a maximum value and a minimumvalue of the representative parallax for each of the stereoscopic imageframes acquired by the representative parallax acquisition unit.

Preferably, when the maximum value of the representative parallaxacquired by the representative parallax acquisition unit is higher thanthe upper limit of the output-allowable parallax width acquired by theoutput-allowable parallax width acquisition unit, the parallax widthadjustment unit adjusts the representative parallax for each of thestereoscopic image frames such that the maximum value of therepresentative parallax falls below the upper limit of theoutput-allowable parallax width.

Preferably, when the minimum value of the parallax acquired by therepresentative parallax acquisition unit falls below the lower limit ofthe output-allowable parallax width acquired by the output-allowableparallax width acquisition unit, the parallax width adjustment unitadjusts the representative parallax for each of the stereoscopic imageframes such that the minimum value of the parallax is higher than thelower limit of the output-allowable parallax width.

Preferably, the reference frame and the target frame are determined froman identical scene.

Preferably, the image output device includes a table acquisition unit toacquire a table that defines a gradual provisional output parallaxcorresponding to a representative parallax with an arbitrary value, andthe provisional output parallax determination unit determines thegradual provisional output parallax for each of the stereoscopic imageframes, in accordance with the representative parallax for each of thestereoscopic image frames acquired by the representative parallaxacquisition unit and the table acquired by the table acquisition unit.

Preferably, the output parallax adjustment unit compares the differencebetween the representative parallax for the reference frame and therepresentative parallax for the target frame with a first predeterminedthreshold value, and, when the difference exceeds the firstpredetermined threshold value, adjusts the output parallax for thetarget frame, to a provisional output parallax that is higher by onegrade than the provisional output parallax for the reference framedetermined by the provisional output parallax determination unit.

Preferably, the output parallax adjustment unit compares the differencewith a second predetermined threshold value, and, when the differencefalls below the second predetermined threshold, adjusts the outputparallax for the target frame, to the provisional output parallax forthe reference frame.

Preferably, when the difference does not exceed the first predeterminedthreshold value and does not fall below the second predeterminedthreshold value, the output parallax adjustment unit adjusts the outputparallax for the target frame, to the provisional output parallax forthe target frame.

Preferably, the first predetermined threshold value and the secondpredetermined threshold value are equal.

The present invention provides an image output method to be executed inan image output device including a representative parallax acquisitionunit, a provisional output parallax determination unit, an outputparallax adjustment unit and an output unit, the image output methodcomprising: a step of acquiring by the representative parallaxacquisition unit a representative parallax for each of multiplestereoscopic image frames constituting a stereoscopic video; a step ofdetermining by the provisional output parallax determination unit aprovisional output parallax for each of the stereoscopic image framesdepending on an output condition of the stereoscopic video, based on therepresentative parallax for each of the stereoscopic image framesacquired by the representative parallax acquisition unit; a step ofadjusting by the output parallax adjustment unit an output parallax foreach of the stereoscopic image frames, based on the provisional outputparallax for each of the stereoscopic image frames determined by theprovisional output parallax determination unit; a step of sequentiallyoutputting by the output unit the stereoscopic image frame whose outputparallax has been adjusted by the output parallax adjustment unit; astep of determining by a provisional output parallax determination unita provisional output parallax for a reference frame based on therepresentative parallax for the reference frame, and determining aprovisional output parallax for a target frame based on therepresentative parallax for the target frame, the reference frame beingsequentially determined from the stereoscopic image frames, the targetframe being the stereoscopic image frame immediately after the referenceframe; and a step of adjusting by the output parallax adjustment unit adifference between the output parallax for the reference frame and theoutput parallax for the target frame, based on a difference between therepresentative parallax for the reference frame and the representativeparallax for the target frame, in which the representative parallax foreach of the stereoscopic image frames includes a statistical operationvalue to be calculated based on parallaxes that, of parallaxes for thestereoscopic image frame, meet a predetermined condition; and thestatistical operation value to be calculated based on the parallaxesthat meet the predetermined condition excludes an average parallax in apredetermined area of the stereoscopic image frame.

The present invention includes also an image output program making theimage output device execute the image output method, and anon-transitory computer-readable recording medium having the programstored therein.

According to the present invention, the difference of the outputparallaxes between stereoscopic image frames is adjusted depending onthe difference of the representative parallaxes between the stereoscopicimage frames, that is, the fluctuation of the representative parallaxes.The output parallax for each stereoscopic image frame is adjusted to anappropriate output parallax, with the fluctuation of the representativeparallaxes at the time of image taking almost kept, allowing for anoutput in which the fluctuation of the representative parallaxes for astereoscopic video is close to the state at the time of image taking.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front perspective diagram of a digital camera.

FIG. 2 is a back perspective diagram of the digital camera.

FIG. 3 is a block diagram of the digital camera.

FIG. 4 is a schematic diagram of a parallax limit in the divergencedirection.

FIG. 5 is a flowchart of a parallax adjustment process.

FIG. 6 is a diagram showing an example of a stereoscopic videoparallax-provisional output parallax conversion table.

FIG. 7 is a flowchart of a parallax width adjustment process.

FIG. 8 is a schematic diagram of a parallax width adjustment.

FIG. 9 is a schematic diagram of a parallax shift in the negativedirection.

FIG. 10 is a schematic diagram of a parallax shift after the parallaxwidth adjustment.

FIG. 11 is a schematic diagram of a parallax shift in the positivedirection.

FIG. 12 is a block diagram of a display playback device.

FIG. 13 is a diagram showing an example of output parallaxes in whichthe fluctuation of parallaxes at the time of image taking is reflected.

FIG. 14 is a diagram showing an example of a conventional parallaxadjustment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 is a front perspective diagram showing the appearanceconfiguration of a digital camera 10 that is an embodiment of thepresent invention. FIG. 2 is a back perspective diagram showing anexample of the appearance configuration of the digital camera.

The digital camera 10 includes multiple sets of imaging means (FIG. 1illustrates two sets.), and allows for an image taking for an identicalsubject from multiple viewpoints (FIG. 1 illustrates two right and leftviewpoints.). Here, in the embodiment, the case including two sets ofimaging means is explained as an example, for the convenience ofexplanation. However, the present invention is not limited to this, andthe case of including three or more sets of imaging means is applicablesimilarly.

A camera body 112 of the digital camera 10 according the embodiment isformed in a rectangular box shape, and as shown in FIG. 1, is providedwith a pair of image-taking optical systems 11R, 11L and a strobe 116 onthe front surface. Furthermore, on the top surface of the camera body112, a release button 14, a power/mode switch 120, a mode dial 122 andthe like are provided. Furthermore, on the back surface of the camerabody 112, as shown in FIG. 2, a monitor 13 configured by a liquidcrystal display device (LCD) and the like, a zoom button 126, a crossbutton 128, a MENU/OK button 130, a DISP button 132, a BACK button 134and the like are provided. The monitor 13 may be incorporated in thedigital camera 10, or may be replaced with an external device (forexample, a TV, a head-mount display or a display of a portableelectronic device such as a mobile phone).

The pair of right and left image-taking optical systems 11R, 11L, isconfigured to include collapsible zoom lenses (18R, 18L in FIG. 3),respectively, and they are extended from the camera body 112 when thepower of the digital camera 10 is turned ON. The zoom mechanism andcollapsing mechanism in the image-taking optical system are knowntechnologies, and therefore the concrete explanation is omitted here.

The monitor 13 is a display device such as a color liquid crystal panelin which a so-called lenticular lens having half-cylindrical lenses isdisposed on the anterior surface. This monitor 13 is utilized as animage display unit for displaying an image after image-taking, and isutilized as a GUI for a variety of setting. Furthermore, it is utilizedas an electronic finder, which through-displays an image picked up by animaging element at the time of image taking. Here, the stereoscopicimage display technique of the monitor 13 is not limited to a parallaxbarrier technique. For example, it may be a stereoscopic image displaytechnique utilizing glasses, such as an anaglyph technique, a polarizingfilter technique or a liquid crystal shutter technique.

The release button 14 is configured as a two-step stroke-type switchthat allows for a so-called “half-push” and “full-push”. When a stillimage is taken (when a still image taking mode is selected through themode dial 122 or a menu), the digital camera 10, once the half-push ofthe release button 14 is performed, performs an image-taking preparationprocess, that is, each process of AE (Automatic Exposure), AF (AutoFocus) and AWB (Automatic White Balance), and once the full push isperformed, it performs an image taking and recording process. When astereoscopic video is taken (when a stereoscopic video taking mode isselected through the mode dial 122 or the menu), once the full push ofthe release button 14 is performed, the stereoscopic video takingstarts, and once the full push is performed again, the image takingfinishes. Here, it may be set such that the stereoscopic video taking isperformed while the full push of the release button 14 is performed, andthe image taking finishes once the full push is released. Furthermore, adedicated release button for taking a still image and a dedicatedrelease button for taking a stereoscopic video may be provided.

The power/mode switch 120 (power switch and mode switch) functions as apower switch of the digital camera 10, and functions as switching meansto switch between a playback mode and an image taking mode of thedigital camera 10. The mode dial 122 is used for the setting in theimage taking mode. The digital camera 10 is set to a 2D still imagetaking mode for taking a 2D still image, by setting the mode dial 122 toa “2D still image position”, and is set to a 3D still image taking modefor taking a 3D still image, by setting the mode dial 122 to a “3D stillimage position”. Furthermore, the digital camera 10 is set to a 3D videotaking mode for taking a 3D video, by setting the mode dial 122 to a “3Dvideo position”.

The zoom button 126 is used for the zoom operation of the image-takingoptical systems 11R, 11L, and is configured by a zoom-tele button forinstructing a zoom to the telescopic side and a zoom-wide button forinstructing a zoom to the wide side. The cross button 128 is providedsuch that the pressing operation can be performed in four directions:rightward, leftward, upward and downward, and functions depending on thesetting state of the camera are assigned to the pressing operations inthe respective directions. The MENU/OK button 130 is used for calling amenu screen (MENU function), and is used for the determination of aselected content, the instruction of a process execution, and the like(OK function). The DISP button 132 is used for inputting a switchinginstruction for the display content on the monitor 13, and the like, andthe BACK button 134 is used for inputting an instruction such as acancel of an input operation.

FIG. 3 is a block diagram showing the principal part of the digitalcamera 10.

The digital camera 10 includes imaging means for the right viewpointthat has the image-taking optical system 11R and an imaging element 29Rfor the right viewpoint, and imaging means for the left viewpoint thathas the image-taking optical system and an imaging element 29L for theleft viewpoint.

The two image-taking optical systems 11 (11R, 11L) each include zoomlenses 18 (18R, 18L), focus lenses 19 (19R, 19L), and apertures 20 (20R,20L). These zoom lenses 18, focus lenses 19 and apertures 20 are drivenby zoom lens control units 22 (22R, 22L), focus lens control units 23(23R, 23L) and aperture control units 24 (24R, 24L), respectively. Therespective control units 22, 23, 24 include stepping motors, and arecontrolled by drive pulses to be given from motor drivers not shown inthe figure, which are connected with a CPU 26.

Behind the two image-taking optical systems 11 (11R, 11L), CCD imagesensors (hereinafter, simply referred to as “CCDs”) 29 (29R, 29L) areeach disposed. Here, MOS-type image sensors may be used instead of theCCDs 29. As is well known, the CCDs 29 include photoelectric conversionsurfaces on each of which multiple photoelectric conversion elements arearrayed, and subject light is entered to the photoelectric conversionsurfaces through the image-taking optical systems 11 so that subjectimages are formed. The CCDs 29 are connected with timing generators: TGs31 (31R, 31L) that are controlled by the CPU 26, and by timing signals(clock pulses) to be input from the TGs 31, the shutter speeds ofelectronic shutters (the charge accumulation time in each photoelectricconversion element) are determined.

The imaging signals output from the CCDs 29 are input to analog signalprocessing circuits 33 (33R, 33L). The analog signal processing circuits33 include correlated double sampling circuits (CDSs), amplifiers (AMPs)and the like. The CDSs generate image data of R, G and B correspondingto the charge accumulation time for each pixel, from the imagingsignals. The AMPs amplify the generated image data.

The AMPs function as sensitivity regulation means to regulate thesensitivities of the CCDs 29. The ISO sensitivities of the CCDs 29 aredetermined by the gains of the AMPs. A/D converters 36 (36R, 36L)convert the amplified image data from analog to digital. The digitalimage data output from the A/D converters 36 (36R, 36L), through imageinput controllers 38 (38R, 38L), are temporarily stored in an SDRAM 39,which is a working memory, as image data for the right viewpoint andimage data for the left viewpoint, respectively.

A digital signal processing unit 41 reads the image data from the SDRAM39, performs various image processes such as a gradation conversion, awhite balance correction, a γ correction process and a YC conversionprocess, and stores the image data in the SDRAM 39 again. The image dataafter the image processing by the digital signal processing unit 41 areacquired by a VRAM 65 as a through image, and thereafter, are convertedinto analog signals for picture output by a display control unit 42, tobe displayed on the monitor 13. The image data after the imageprocessing that are acquired following the full-push of the releasebutton 14 are compressed in a predetermined compression format (forexample, the MEG format) by a compression and decompression processingunit 43, and thereafter, through a medium control unit 15, are recordedin a memory card 16 as recording image.

An operation unit 25, by which various operations for the digital camera10 are performed, is configured by the various buttons and switches 120to 134 shown in FIG. 1 and FIG. 2.

The CPU 26 is provided for integrally controlling the digital camera 10.The CPU 26 controls the respective units such as a battery 70, a powercontrol unit 71 and a watch unit 72, based on various control programsand setting information stored in non-transitory computer-readablerecording media such a flash ROM 60 and a ROM 61, input signals from anattitude detection sensor 73 and the operation unit 25, and the like.

Furthermore, the digital camera 10 is provided with an AE/AWB controlunit 47 to perform an AE (Auto Exposure)/AWB (Auto White Balance)control, and a parallax detection unit 49 to detect a representativeparallax for each of multiple stereoscopic image frames (hereinafter,also simply referred to as “frames”). Furthermore, the digital camera 10includes a flash control unit 23 to control the luminescence timing andluminescence amount of a flash 5.

When the half-push of the release button 14 is performed, the AE/AWBcontrol unit 47 analyzes images (pickup images) obtained by the CCDs 29,and calculates the aperture values of the apertures 20 and the shutterspeeds of the electronic shutters of the CCDs 29, based on the luminanceinformation of a subject and the like. Then, based on these calculationresults, the AE/AWB control unit 47 controls the aperture values throughthe aperture control units 24, and controls the shutter speeds throughTGs 31.

For example, based on a pickup image (right viewpoint image or leftviewpoint image) obtained by the CCD 29R or CCD 29L of one image-takingoptical system of the two image-taking optical systems 11R, 11L, theaperture values and shutter speeds of both image-taking optical systems11R, 11L are calculated. Based on pickup images (right viewpoint imageand left viewpoint image) obtained by both image-taking optical systems11R and 11L, the respective aperture values and shutter speeds of theimage-taking optical systems 11R, 11L may be calculated.

When the half-push of the release button 14 is performed, an AF controlunit 45 performs an AF search control of moving the focus lenses 19R,19L in the optical axis direction and calculating the contrast value,and performs a focusing control of moving the focus lenses 19R, 19L tothe focusing lens positions based on the contrast value. Here, the“contrast value” is calculated based on image signals in predeterminedfocusing evaluation value calculation areas of pickup images obtained bythe CCDs 29R, 29L. The “focusing lens positions” are positions of thefocus lenses 19R, 19L where the focus lenses 19R, 19L are focused on atleast a main subject.

For example, while at least one of the focus lenses 19R, 19L of the twoimage-taking optical systems 11R, 11L is moved by the drive of the focuslens control unit 23R or 23L, the contrast value is calculated from apickup image (right viewpoint image or left viewpoint image) by one ofthe image-taking optical systems 11R and 11L. Based on the contrastvalue, the focusing lens positions of the focus lenses 19R, 19L of thetwo image-taking optical systems 11R, 11L are determined respectively,and by the drive of the respective focus lens control units 23R and 23L,the focus lenses 19R, 19L are moved to the respective focusing lenspositions. The respective focusing lens positions may be determined byperforming the AF search with both image-taking optical systems 11R,11L, respectively.

The attitude detection sensor 73 detects the rotation directions andangles of the image-taking optical systems 11R, 11L relative topreviously determined attitudes.

A camera shake control unit 62 drives, with motors, correction lensesnot shown in the figure which are provided in the image-taking opticalsystems 11R, 11L, and thereby, corrects optical axis gaps detected bythe attitude detection sensor 73 to prevent camera shake.

The CPU 26 controls a face recognition unit 64 such that a facerecognition is performed from the right and left image datacorresponding to the subject images in the image-taking optical systems11R, 11L. In response to the control of the CPU 26, the face recognitionunit 64 starts a face recognition, and performs the face recognitionfrom each of the right and left image data. The face recognition unit 64stores, in the SDRAM 39, the face area information containing theposition information of face areas, which are respectively recognizedfrom the right and left image data as a result of the face recognition.By a known method such as template matching, the face recognition unit64 can recognize the face area from the images stored in the SDRAM 39.Here, examples of the face area of a subject include the face area of aperson or animal in a pickup image.

A face correspondence decision unit 66 decides the correspondencerelation between the face area recognized from the right image data andthe face area recognized from the left image data. That is, the facecorrespondence decision unit 66 identifies a pair of face areas, forwhich the pieces of position information of the face areas respectivelyrecognized from the right and left image data are closest to each other.Then, the face correspondence decision unit 66 performs the matching ofthe image information between the face areas constituting the pair, anddecides that the face areas constituting the pair have a correspondencerelation to each other, when the accuracy of both the identity exceeds apredetermined threshold value.

The parallax detection unit 49 calculates a representative parallaxbetween predetermined areas of the right and left image data. Thepredetermined area can include a partial area or whole area of an image.Furthermore, the predetermined area can include a face area detected bya known face detection, a face area matching a face area of a particularperson that is arbitrarily registered, a gazing point of an observer ofthe monitor 13 or a videographer, a gazing point or subject area of astereoscopic video in the display surface of the monitor 13 that isarbitrarily designated through a user interface such as the operationunit 25, a vicinity area of a gazing point, or the like. When there is asingle observer for the monitor 13, the gazing point of the observer canbe detected by a known gazing point detection such as PTLs 2 to 4. Whenthere are multiple observers for the monitor 13, a known gazing pointdetection is applied to an observer meeting a particular condition, forexample, the closest observer to the monitor 13, and thereby, the gazingpoint can be detected. The gazing point of a videographer may be a wholeor part of a subject area that is designated at the time of image takingor at another timing. The gazing point detection may be performed by thedigital camera 10, or may be performed by another device, for example,an external display device such as a television or a head-mount display.

For example, the representative parallax is calculated as follows.First, the parallax detection unit 49 calculates a position difference(corresponding-point distance) between particular points (correspondingpoints) that correspond between face areas constituting a pair. Then,the parallax detection unit 49 calculates the average value of theparallaxes for the points contained in the pair of face areas, and thisis determined as the representative parallax for the pair. When multipleface areas are decided as having correspondence relations, the parallaxdetection unit 49 calculates the representative parallax only for a mainface area of those face areas, and stores the representative parallaxfor the main face area in the SDRAM 39. The main face area is a facearea closest to the center of the screen, a face area closest to thefocusing evaluation value calculation area, a face area having thelargest size, or the like.

Alternatively, the parallax detection unit 49 makes a parallax histogramfrom the parallaxes for the corresponding points in a predetermined area(for example, a partial area such as a face area, or the whole area) ofthe right and left image data. The class is arbitrary. Then, the classvalue of a class having the highest frequency, that is, the mode valuemay be determined as the representative parallax for the predeterminedarea.

Alternatively, the parallax detection unit 49 selects the maximumparallax value or minimum parallax value from the parallaxes for thecorresponding points in a predetermined area of the right and left imagedata, and may determine the maximum parallax value or minimum parallaxvalue as the representative parallax for the predetermined area. Here,when the predetermined area is a single point such as a gazing point,the representative parallax is the maximum parallax value and theminimum parallax value for the gazing point.

Alternatively, the parallax detection unit 49 may determine the medianvalue of the parallaxes for the corresponding points in a predeterminedarea of the right and left image data, as the representative parallaxfor the predetermined area.

Alternatively, the parallax detection unit 49 may determine the averagevalue of the parallaxes for the corresponding points in a predeterminedarea of the right and left image data, as the representative parallaxfor the predetermined area.

Other than the above, the representative parallax can be calculated byvarious statistical operations. Here, values that do not meet apredetermined condition may be excluded from sample values of theparallaxes to be used for calculating the representative parallax. Forexample, the parallaxes for the corresponding points in an area wherethe sharpness (spatial frequency) of the image is lower than apredetermined reference value (a so-called defocus area), the parallaxesexceeding a predetermined limit value, and the like may be excluded fromthe sample values of the parallaxes to be used for calculating therepresentative parallax, and the representative parallax may becalculated from the parallaxes for the corresponding points in an areameeting the predetermined condition. The condition may be determinedbased on image-taking conditions such as the zoom factors of theimage-taking optical systems 11R, 11L. This is because the parallax foran identical corresponding point is magnified or demagnified in responseto a magnification or demagnification of the zoom factor. Alternatively,there may be no predetermined exclusion condition (no condition).

Alternatively, using the above statistical operations, the parallaxdetection unit 49 may calculate the representative parallax, from theparallaxes for a subject positioned on the far side (background side),which is a side in the direction farther from the digital camera 10 thana cross point of the right and left image data, or on the near side(foreground side), which is a side in the direction closer to thedigital camera 10. Here, the cross point is a convergent point at whichthe optical axis of the image-taking optical system 11R and the opticalaxis of the image-taking optical system 11L intersect on theimage-taking symmetry plane.

Furthermore, the representative parallax may be calculated for differentframes or over different scenes, by a common statistical operationexpression, or multiple kinds of representative parallaxes may becalculated from an identical frame by multiple statistical operationexpressions.

For example, the maximum value of the parallaxes for one arbitrary frameF is determined as a first representative parallax, and the minimumvalue of the parallaxes for the frame F is determined as a secondrepresentative parallax. Thus, multiple kinds of representativeparallaxes may be determined from one frame.

Alternatively, as for the representative parallax for each frameconstituting a background scene X, using the above statisticaloperations, the representative parallax may be calculated from theparallaxes for the corresponding points of a subject that is positionedon the far side (background side) relative to the cross point. Then, asfor the representative parallax for each frame constituting a foregroundscene Y, using the above statistical operations, the representativeparallax may be calculated from the parallaxes for the correspondingpoints of a subject that is positioned on the near side (foregroundside) relative to the cross point. For example, for one arbitrary frame,the maximum value of the parallaxes for the corresponding points on thebackground side relative to the cross point can be determined as a firstrepresentative parallax, and the maximum value of the parallaxes for thecorresponding points on the foreground side relative to the cross pointcan be determined as a second representative parallax.

Alternatively, the maximum value of the parallaxes for the correspondingpoints on the background side relative to the cross point can bedetermined as a first representative parallax, the maximum value of theparallaxes for the corresponding points on the foreground side relativeto the cross point can be determined as a second representativeparallax, the minimum value of the parallaxes for the correspondingpoints on the background side relative to the cross point can bedetermined as a third representative parallax, and the minimum value ofthe parallaxes for the corresponding points on the foreground siderelative to the cross point can be determined as a forth representativeparallax.

Alternatively, the maximum value of all the parallaxes for an arbitraryframe contained in an identical scene can be determined as a firstparallax, and the minimum value of all the parallaxes can be determinedas a second parallax.

That is, only a single kind of representative parallax may be determinedby a single statistical operation, or multiple kinds of representativeparallaxes may be determined by multiple different statisticaloperations.

Alternatively, the parallax detection unit 49 calculates the averagevalue of the parallaxes between the corresponding points inpredetermined areas that have a correspondence relation between theright and left images, for example, the center areas of the images orfocusing evaluation value calculation areas, and determines it as therepresentative parallax for the pair.

The position information and representative parallax for predeterminedareas that have a correspondence relation are stored in the SDRAM 39while being associated with the right and left image data. For example,the position information and representative parallax for face areas thathave a correspondence relation are stored as the supplementaryinformation (header, tag, meta-information or the like) of the imagedata. In the case where the image data is compressed and recorded in thememory card 16 as recording images, the position information andrepresentative parallax for the face areas are recorded together in thesupplementary information of the recording images, as the taginformation such as Exif, for example.

A display-allowable parallax width acquisition unit 204 acquires aminimum display-allowable parallax Dmin and a maximum display-allowableparallax Dmax, and inputs them to a parallax width adjustment unit 202.The mode of the acquisition is arbitrary, and they may be input from theoperation unit 25, may be input from the ROM 61, the supplementaryinformation of stereoscopic video data, or the like, or may be inputfrom the monitor 13 as control information.

The maximum display-allowable parallax Dmax defines the limit of theparallax in the divergence direction (in the direction in which thestereoscopic image on the monitor 13 goes backward). As illustrated inthe (a) portion of FIG. 4, since human eyes do not spread outward, rightand left images having a parallax beyond the distance between the pupilsdo not fuse, and a viewer cannot recognize them as one image, causingeyestrain. Considering a child viewer, the distance between the pupilsis about 5 cm, and therefore, the number of pixels of the monitor 13corresponding to the distance is the maximum display-allowable parallaxDmax. For example, when the monitor 13 is a high-vision televisionhaving a size of 16:9 inches and the resolution is 1920×1080, themaximum display-allowable parallax Dmax for each size of the monitor 13is as shown in the (b) portion of FIG. 4. In the case where the size ofthe monitor 13 is small as a built-in screen of a digital camera ormobile phone, the parallax in the divergence direction hardly becomes aproblem, but in the case of the monitor 13 having a large-size displaysurface as a television, the parallax in the divergence directionbecomes a problem.

The minimum display-allowable parallax Dmin defines the limit of theexcessive parallax (in the direction in which the stereoscopic image onthe monitor 13 goes forward). Unlike the maximum display-allowableparallax Dmax, the minimum display-allowable parallax Dmin cannot beuniquely determined from the distance between the pupils. As outputconditions for determining the minimum display-allowable parallax Dmin,for example, there are (1) the size of the monitor 13, (2) theresolution of the monitor 13, (3) the viewing distance (the distancefrom a viewer to the monitor 13), and (4) the individual stereoscopicfusion limit of a viewer.

As a standard example, (2) the resolution of the monitor 13 of ahigh-vision television is 1920×1080, and (3) the viewing distance isthree times the screen height of the monitor 13. Assuming these, (4) ageneral stereoscopic fusion limit is 57 pixels (a parallax angle ofabout 1 degree). A threshold value setting unit 205 may input theinformation of (1) to (4) from the exterior, based on a user operation,the setting information of the monitor 13, or the like. For example, auser can input the resolution of the monitor 13 that the user isviewing, the viewing distance, the stereoscopic fusion limit, throughthe operation unit 25. However, when (2) to (4) are not particularlyinput from the exterior, the threshold value setting unit 205 reads theabove typical example from the ROM 61 or the like, and then input it tothe parallax width adjustment unit 202.

The parallax width adjustment unit 202 adjusts the width of therepresentative parallaxes for the right and left image data such that itfalls within the display-allowable parallax width ranging from theminimum display-allowable parallax Dmin to the maximum display-allowableparallax Dmax.

The threshold value setting unit 205 sets a shift-allowable thresholdvalue α and a shift-prohibitive threshold value to the parallaxadjustment unit 63. The setting manner, which is arbitrary, includes amanner based on a user operation, a manner based on the recordedinformation in the ROM 61, and the like. The parallax adjustment unit 63adjusts the value of the representative parallax for the right and leftimage data, in accordance with the shift-allowable threshold value α,the shift-prohibitive threshold value β, a stereoscopic videoparallax-provisional output parallax conversion table described later.

FIG. 5 is a flowchart showing the parallax adjustment process. Thisprocess is controlled by the CPU 26. A program making the CPU 26 executethis process is stored in a computer-readable recording medium such asthe ROM 61. This process is executed after the position information andrepresentative parallax for the above areas are stored in thesupplementary information of image data.

In S1, the parallax width adjustment unit 202 performs a parallax widthadjustment process described later. In the parallax adjustment process,the adjustment of the parallax width of the representative parallaxesand the shift of the representative parallaxes are performed, asnecessary.

In S2, the parallax adjustment unit 63 secures the representativeparallax for each stereoscopic image frame after the parallax widthadjustment process, in the SDRAM 39.

Then, the parallax adjustment unit 63 determines a reference frame thatis a stereoscopic image frame of reference for the parallax adjustment.The reference frame is determined in accordance with the temporalacquisition order for the stereoscopic images. For example, in an n-thtime execution of a loop of S2 to S12, the parallax adjustment unit 63determines the reference frame as an n-th stereoscopic image frame inthe acquisition order. The parallax adjustment unit 63 secures areference parallax that is a representative parallax corresponding tothe reference frame, in the SDRAM 39.

In S3, the parallax adjustment unit 63 determines a target frame that isa stereoscopic image frame as a target of the parallax adjustment. Forexample, in the n-th time execution of the loop of S2 to S12, theparallax adjustment unit 63 determines an n+1-th stereoscopic imageframe in the acquisition order as the target frame. The parallaxadjustment unit 63 secures a representative parallax corresponding tothe target frame, in the SDRAM 39.

In S4, the parallax adjustment unit 63 decides whether |therepresentative parallax for the reference frame−the representativeparallax for the target frame|<α holds. In the case of Yes, the processproceeds to S5, and in the case of No, the process proceeds to S6. Asthe shift-allowable threshold value, α is input from the threshold valuesetting unit 205. For example, α is 0.75. The decision of No means thatthe variation in the representative parallax between the stereoscopicimage frames is large to some extent. In this case, the process proceedsto S6, and a process for shifting the target frame to an output parallaxdifferent from that of the reference frame is performed.

In S5, the parallax adjustment unit 63 decides whether |therepresentative parallax for the reference frame−the representativeparallax for the target frame|>β holds. In the case of Yes, the processproceeds to S10, and in the case of No, the process proceeds to S8. Asthe shift-prohibitive threshold value, β is input from the thresholdvalue setting unit 205. For example, β is 0.25. The decision of No meansthat the variation in the representative parallax between thestereoscopic image frames is small. In this case, the process proceedsto S8, and a process for shifting the target frame to an output parallaxequivalent to that of the reference frame is performed.

In S6, the parallax adjustment unit 63 reads out the stereoscopic videoparallax-provisional output parallax conversion table stored in the ROM61 or the like, to the SDRAM 39. FIG. 6 shows an example of thestereoscopic video parallax-provisional output parallax conversiontable. This table defines a provisional output parallax having aninteger that corresponds to a representative parallax for eachstereoscopic image frame, which is an arbitrary value. In this table,for example, representative parallaxes of M to M+t correspond to aprovisional output parallax of N, and representative parallaxes of M+tto M+2t correspond to a provisional output parallax of N+1. Here, sincethe minimum display unit of an image is 1 pixel, the provisional outputparallax in the pixel unit is shown as an integer.

In accordance with the stereoscopic video parallax-provisional outputparallax conversion table stored in the ROM 61 or the like, the parallaxadjustment unit 63 specifies a provisional output parallax correspondingto the representative parallax for the reference frame, and determinesthat the specified provisional output parallax is the provisional outputparallax for the reference frame. Similarly, in accordance with thestereoscopic video parallax-provisional output parallax conversiontable, the parallax adjustment unit 63 specifies a provisional outputparallax corresponding to the representative parallax for the targetframe, and determines that the specified provisional output parallax isthe provisional output parallax for the target frame.

The parallax adjustment unit 63 compares the provisional output parallaxfor the reference frame and the provisional output parallax for thetarget frame, and decides whether both are equivalent. In the case ofYes, the process proceeds to S7, and in the case of No, the processproceeds to S10.

In S7, the parallax adjustment unit 63 determines that the outputparallax for the target frame is the provisional output parallax for thereference frame+1, and performs a parallax adjustment of shifting therepresentative parallax for the target frame to this output parallax(the provisional output parallax for the reference frame+1). That is,even if the provisional output parallaxes for the reference frame andthe target frame are equivalent, when the variation in therepresentative parallax between the reference frame and the target frameis large, the output parallaxes for both are segregated so that thefluctuation from the original of the representative parallax isreflected in the output parallax. Thereafter, the process proceeds toS11.

In S8, a similar decision to S6 is performed. In the case of Yes, theprocess proceeds to S10, and in the case of No, the process proceeds toS9.

In S9, the parallax adjustment unit 63 determines that the outputparallax for the target frame is the provisional output parallax for thereference frame, and performs a parallax adjustment of shifting therepresentative parallax for the target frame to this output parallax(the provisional output parallax for the reference frame). That is, evenif the provisional output parallaxes for the reference frame and thetarget frame are different, when the variation in the representativeparallax between the reference frame and the target frame is small, theoutput parallax for the target frame is equalized to the provisionaloutput parallax for the reference frame so that the fluctuation from theoriginal of the representative parallax is reflected in the outputparallax. Thereafter, the process proceeds to S11.

In S10, the parallax adjustment unit 63 determines that the outputparallax for the target frame is the provisional output parallax for thetarget frame, and adjusts the parallax for the target frame to thisoutput parallax (the provisional output parallax for the target frame).As patterns of proceeding from S4 to S10, there are three patterns: (a)Yes in S4 and Yes in S5, (b) Yes in S4 and No in S5 and Yes in S8, and(c) No in S4 and No in S6. In the (b) or the (c), the largeness of theoriginal variation in the representative parallax between the referenceframe and the target frame is reflected in the largeness of thevariation in the output parallax, with no change. The (a) is a method ofleaving, to the provisional output parallax, the adjustment of theoutput parallax for the target frame corresponding to an intermediatevariation in which the original variation in the representative parallaxis not large and not small. Here, in the case of α=β, the (a) patterndoes not occur logically, and the process becomes simpler. For example,α=β=0.5 is possible.

In S11, whether S2 to S10 have been executed for all the stereoscopicimage frames constituting the stereoscopic video and the parallaxadjustment has been executed for all the stereoscopic image frames isdecided. In the case of Yes, the process proceeds to S13, and in thecase of No, the process proceeds to S12.

In S12, the parallax adjustment unit 63 determines that the referenceframe is the n+1-th stereoscopic image frame.

In S13, the display control unit 42 sequentially displays, on themonitor 13, the stereoscopic image frames at the adjusted outputparallaxes, and thereby, plays back the stereoscopic video. Here, in S7,S9 and S10, the parallax adjustment unit 63 performs a parallaxadjustment for shifting the output parallax for the first referenceframe to the provisional output parallax for the reference frame. Thisis because, although the parallax adjustment as the target frame isperformed for the second and subsequent reference frames, this parallaxadjustment is not performed for the first reference frame.

FIG. 7 shows a flowchart of the parallax width adjustment process.

In S101, the parallax width adjustment unit 202 attempts reading therepresentative parallax for each stereoscopic image frame, from theright and left image data for each stereoscopic image frame of thestereoscopic video and the supplementary information of the stereoscopicvideo that are stored in the SDRAM 39 or the memory card 16.

In S102, the display-allowable parallax width acquisition unit 204acquires the display-allowable parallax width to the SDRAM 39. Thedisplay-allowable parallax width is the range from the minimumdisplay-allowable parallax Dmin to the maximum display-allowableparallax Dmax. The acquisition source of the display-allowable parallaxwidth includes the operation unit 25, the built-in ROM 61, the externalmonitor 13, electronic devices and the like.

In S103, the parallax width adjustment unit 202 specifies the maximumvalue pmax of the representative parallax and the minimum value pmin ofthe representative parallax, from the representative parallax for eachstereoscopic image, and calculates a stereoscopic video parallax widthby using pmax−pmin. Then, the parallax width adjustment unit 202 decideswhether the stereoscopic video parallax width<the display-allowableparallax width holds. In the case of Yes, the process proceeds to S105,and in the case of No, the process proceeds to S104.

In S104, the parallax width adjustment unit 202 adjusts therepresentative parallax for each stereoscopic image frame such that thestereoscopic video parallax width falls within the display-allowableparallax width. For example, when the stereoscopic video parallax widthexceeds the display-allowable parallax width as shown in the (a) portionof FIG. 8, the representative parallax for each stereoscopic image frameis uniformly reduced by a reduction ratio of (X−Y)/X, such that thestereoscopic video parallax width falls within the range of thedisplay-allowable parallax width, as shown in the (b) portion of FIG. 8.

In S105, the parallax width adjustment unit 202 decides whether themaximum value pmax of the representative parallax>the maximumdisplay-allowable parallax Dmax holds. In the case of Yes, the processproceeds to S107, and in the case of No, the process proceeds to S106.

In S106, the parallax width adjustment unit 202 decides whether theminimum value pmin of the representative parallax<the minimumdisplay-allowable parallax Dmin holds. In the case of Yes, the processproceeds to S107, and in the case of No, the process proceeds to S2 ofthe parallax adjustment process.

In S107, the parallax width adjustment unit 202 shifts therepresentative parallax for each stereoscopic image frame such that thestereoscopic video parallax width falls within the display-allowableparallax width.

As patterns of proceeding from S103 to S107, there are four patterns:(1) Yes in S103 and Yes in S105, (2) No in S103 and Yes in S105, (3) Yesin S103 and No in S105 and Yes in S106, and (4) No in S103 and No inS105 and Yes in S106.

FIG. 9 shows Pattern (1), that is, a shift in the negative directionwhen the parallax width adjustment is not performed.

For example, when the maximum value pmax of the representative parallaxexceeds the maximum display-allowable parallax Dmax but the stereoscopicvideo parallax width is less than the display-allowable parallax widthas shown in the (a) portion of FIG. 9, the representative parallax foreach stereoscopic image frame is shifted in the negative direction by auniform width W1, and an adjustment is performed such that therepresentative parallaxes for all the stereoscopic image frames fallwithin the range of the display-allowable parallax width, as shown inthe (b) portion of FIG. 9. Here, W1 is pmin−Dmin.

FIG. 10 shows Pattern (2), that is, a shift in the negative directionwhen the parallax width adjustment is performed.

When the maximum value pmax of the representative parallax after theparallax width adjustment exceeds the maximum display-allowable parallaxDmax as shown in the (a) portion of FIG. 10 and the (b) portion of FIG.8 described above, also, the representative parallax for eachstereoscopic image frame is shifted in the negative direction by auniform width W2 as shown in the (b) portion of FIG. 10. Here, W2 ispmin−Dmin.

FIG. 11 shows Pattern (3), that is, a shift in the positive directionwhen the parallax width adjustment is not performed.

Alternatively, when the minimum value pmin of the representativeparallax falls below the minimum display-allowable parallax Dmin asshown in the (a) portion of FIG. 11, the representative parallax foreach stereoscopic image frame is shifted in the positive direction by auniform width W3 as shown in the (b) portion of FIG. 11. Here,W3=Dmin−pmin.

A figure of Pattern (4) is omitted. When the minimum value pmin of therepresentative parallax after the parallax width adjustment falls belowthe minimum display-allowable parallax Dmin, the representative parallaxfor each stereoscopic image frame, similarly, is shifted in the positivedirection by a uniform width.

Here, the above parallax adjustment process is repeated until a changein scenes is detected. When a change in scenes is detected, thereference frame and the target frame are reset, and therefrom, S1 startsanew. Thereby, it is possible to make the decisions in S4, 5, 6 and 8between a reference frame and target frame that are across differentscenes, and to prevent an appropriate parallax adjustment. The detectionof a change in scenes is performed by a known method. A change in scenesoccurs by a change in focused subjects, a panning and the like.

For example, when the representative parallaxes between differentstereoscopic image frames a and b are equal to or greater than athreshold value, the parallax adjustment unit 63 detects a scene changebetween the stereoscopic image frames a and b. When the stereoscopicimage frame a immediately before the scene change is set as the lasttarget frame, the parallax adjustment unit 63 decides No in S11, and theprocess proceeds to S12. However, in S12, the parallax adjustment unit63 does not determine that the reference frame is the stereoscopic imageframe a immediately before the scene change, and determines that thereference frame is the stereoscopic image frame b immediately after thescene change. Furthermore, the parallax adjustment unit 63 determinesthat the target frame is a stereoscopic image frame c following thestereoscopic image frame b. Then, similarly, the parallax adjustmentunit 63 repeats the loop of S2 to S12 from a stereoscopic image frameimmediately after a previous scene change to a stereoscopic image frameimmediately before the next scene change.

Furthermore, the blocks necessary for executing the process may beincluded in an electronic device other than a digital camera. Forexample, as shown in FIG. 12, the process can be executed by an imageoutput device including blocks for displaying a planar or stereoscopicimage, such as the CPU 26, the VRAM 65, the SDRAM 39, the flash ROM 60,the ROM 61, the compression and decompression processing unit 43, themedium control unit 15, the parallax detection unit 49, the parallaxadjustment unit 63, the image input unit 201 (for example, the imageinput controller 38, the medium control unit 15 and the like), thedisplay-allowable parallax width acquisition unit 204, the thresholdvalue setting unit 205, and an image output unit 206 (for example, themonitor 13, the medium control unit 15 and the like).

A stereoscopic video to be input by the image input unit 201 is notlimited to a stereoscopic video directly output from the imaging means,and for example, may be a stereoscopic video that the medium controlunit 15 reads from a medium such as the memory card 16, or that isreceived through a network.

The destination to which the image output unit 206 outputs an imageafter the parallax adjustment completion is not limited to the displaycontrol unit 42 and the monitor 13, and the image does not need to bedisplayed promptly after the parallax adjustment. For example, themedium control unit 15 may store the adjusted representative parallaxfor each stereoscopic image frame, that is, the output parallax, in amedium such as the memory card 16, as stereoscopic video data associatedwith each stereoscopic image frame. Alternatively, the stereoscopicvideo data may be sent through a network.

Also, the mode setting of whether the parallax adjustment process isoperated, and the timing therefor are arbitrary. For example, when theimage taking mode is started, the parallax adjustment process is notperformed, and after a full-push of the release button 14 is performed,the parallax adjustment is started. Alternatively, when stereoscopicvideo data of the memory card 16 are displayed on the external monitor13 such as a television, the parallax adjustment is started.

By the above process, the representative parallax for each stereoscopicimage frame is adjusted to an appropriate output parallax, with thefluctuation (see the (a) portion of FIG. 13) of the parallaxes at thetime of image taking almost kept (see the (b) portion of FIG. 13). Thisallows for a playback in which the fluctuation of the representativeparallaxes for a stereoscopic video is close to the state at the time ofimage taking.

Here, the above parallax adjustment process and parallax widthadjustment process can be executed for each of the different kinds ofrepresentative parallaxes. In such case, there is a possibility that theparallax adjustment results and the parallax width adjustment resultsare not consistent among the different kinds of representativeparallaxes. On this occasion, results appropriate for an observer may beselected and executed, or the parallax adjustment and the parallax widthadjustment may be cancelled.

For example, suppose that, in one arbitrary frame, the maximum value ofthe parallaxes on the background side relative to the cross point is afirst representative parallax, and the maximum value of the parallaxeson the foreground side relative to the cross point is a secondrepresentative parallax, and that the parallax width adjustment processis performed for the first representative parallax, and the parallaxwidth adjustment process is performed for the second representativeparallax.

In this case, even if, by setting a value exceeding 1 as a zoom factor,YES is decided in S103 for the second representative parallax, theprocess may proceed to S104 if NO is decided in S103 for the firstrepresentative parallax.

Alternatively, even if NO is decided in S105 and NO is decided in S106for the second representative parallax, the process may proceed to S107if YES is decided in S105 or YES is decided in S106 for the firstrepresentative parallax. Thereby, it is possible to balance the parallaxadjustment on the background side and the parallax adjustment on theforeground side.

What is claimed is:
 1. An image output device comprising: arepresentative parallax acquisition unit to acquire a representativeparallax for each of multiple stereoscopic image frames constituting astereoscopic video; a provisional output parallax determination unit todetermine a provisional output parallax for each of the stereoscopicimage frames depending on an output condition of the stereoscopic video,based on the representative parallax for each of the stereoscopic imageframes acquired by the representative parallax acquisition unit; anoutput parallax adjustment unit to adjust an output parallax for each ofthe stereoscopic image frames, based on the provisional output parallaxfor each of the stereoscopic image frames determined by the provisionaloutput parallax determination unit; and an output unit to sequentiallyoutput each of the stereoscopic image frames whose output parallax isadjusted by the output parallax adjustment unit, wherein the provisionaloutput parallax determination unit determines the provisional outputparallax for a reference frame corresponding to the representativeparallax for the reference frame, and determines the provisional outputparallax for a target frame corresponding to the representative parallaxfor the target frame, in accordance with the provisional output parallaxdefined for each range of the representative parallax for thestereoscopic image frame, the reference frame being sequentiallydetermined from the stereoscopic image frames, and the target framebeing a stereoscopic image frame immediately after the reference frame;the output parallax adjustment unit adjusts a difference between theoutput parallax for the reference frame and the output parallax for thetarget frame, based on a difference between the representative parallaxfor the reference frame and the representative parallax for the targetframe; the representative parallax for each of the stereoscopic imageframes includes a statistical operation value to be calculated based onparallaxes that, of parallaxes for the stereoscopic image frame, meet apredetermined condition; and the statistical operation value to becalculated based on the parallaxes that meet the predetermined conditionexcludes an average parallax in a predetermined area of the stereoscopicimage frame.
 2. The image output device according to claim 1, whereinthe statistical operation value includes at least one of a maximumvalue, a minimum value, a mode value and a median value of theparallaxes for the stereoscopic image frame.
 3. The image output deviceaccording to claim 2, wherein the representative parallax for eachstereoscopic image frame includes at least one of a maximum value, aminimum value, a mode value and a median value of parallaxes for asubject present on a background side or parallaxes for a subject presenton a foreground side, among the parallaxes for the stereoscopic imageframe, the background side being a side in a direction farther from animaging device than a cross point, and the foreground side being a sidein a direction closer to the imaging device than the cross point.
 4. Theimage output device according to claim 1, wherein the parallaxes thatmeet the predetermined condition include a parallax for a gazingposition to the stereoscopic image frame.
 5. The image output deviceaccording to claim 4, wherein the gazing position includes a gazingpoint of a viewer of the stereoscopic image frame, a gazing point of avideographer of the stereoscopic image frame, or an arbitrary designatedgazing point in the stereoscopic image frame.
 6. The image output deviceaccording to claim 1, wherein the parallaxes that meet the predeterminedcondition include parallaxes for a face area, parallaxes for a focusingevaluation value calculation area, parallaxes for a central image area,parallaxes for a subject present on a background side among theparallaxes for the stereoscopic image frame, or parallaxes for a subjectpresent on a foreground side among the parallaxes for the stereoscopicimage frame, the background side being a side in a direction fartherfrom an imaging device than a cross point, and the foreground side beinga side in a direction closer to the imaging device than the cross point.7. The image output device according to claim 1, further comprising anoutput-allowable parallax width acquisition unit to acquire an upperlimit and a lower limit defining an output parallax width, as the outputcondition of the stereoscopic video, the output parallax width being awidth of the output parallax that is allowable.
 8. The image outputdevice according to claim 7, further comprising a parallax widthadjustment unit to adjust a parallax width to the output-allowableparallax width acquired by the output-allowable parallax widthacquisition unit, when the parallax width does not comply with theoutput-allowable parallax width, the parallax width being defined by amaximum value and a minimum value of the representative parallax foreach of the stereoscopic image frames acquired by the representativeparallax acquisition unit.
 9. The image output device according to claim8, wherein, when the maximum value of the representative parallaxacquired by the representative parallax acquisition unit is higher thanthe upper limit of the output-allowable parallax width acquired by theoutput-allowable parallax width acquisition unit, the parallax widthadjustment unit adjusts the representative parallax for each of thestereoscopic image frames such that the maximum value of therepresentative parallax falls below the upper limit of theoutput-allowable parallax width.
 10. The image output device accordingto claim 8, wherein, when the minimum value of the parallax acquired bythe representative parallax acquisition unit falls below the lower limitof the output-allowable parallax width acquired by the output-allowableparallax width acquisition unit, the parallax width adjustment unitadjusts the representative parallax for each of the stereoscopic imageframes such that the minimum value of the parallax is higher than thelower limit of the output-allowable parallax width.
 11. The image outputdevice according to claim 1, wherein the reference frame and the targetframe are determined from an identical scene.
 12. The image outputdevice according to claim 1, further comprising a table acquisition unitto acquire a table that defines a gradual provisional output parallaxcorresponding to a representative parallax with an arbitrary value,wherein the provisional output parallax determination unit determinesthe gradual provisional output parallax for each of the stereoscopicimage frames, in accordance with the representative parallax for each ofthe stereoscopic image frames acquired by the representative parallaxacquisition unit and the table acquired by the table acquisition unit.13. The image output device according to claim 12, wherein the outputparallax adjustment unit compares the difference between therepresentative parallax for the reference frame and the representativeparallax for the target frame with a first predetermined thresholdvalue, and, when the difference exceeds the first predeterminedthreshold value, adjusts the output parallax for the target frame, to aprovisional output parallax that is higher by one grade than theprovisional output parallax for the reference frame determined by theprovisional output parallax determination unit.
 14. The image outputdevice according to claim 13, wherein the output parallax adjustmentunit compares the difference with a second predetermined thresholdvalue, and, when the difference falls below the second predeterminedthreshold, adjusts the output parallax for the target frame, to theprovisional output parallax for the reference frame.
 15. The imageoutput device according to claim 14, wherein, when the difference doesnot exceed the first predetermined threshold value and does not fallbelow the second predetermined threshold value, the output parallaxadjustment unit adjusts the output parallax for the target frame, to theprovisional output parallax for the target frame.
 16. The image outputdevice according to claim 15, wherein the first predetermined thresholdvalue and the second predetermined threshold value are equal.
 17. Animage output method to be executed in an image output device including arepresentative parallax acquisition unit, a provisional output parallaxdetermination unit, an output parallax adjustment unit and an outputunit, the image output method comprising: a step of acquiring by therepresentative parallax acquisition unit a representative parallax foreach of multiple stereoscopic image frames constituting a stereoscopicvideo; a step of determining by the provisional output parallaxdetermination unit a provisional output parallax for each of thestereoscopic image frames depending on an output condition of thestereoscopic video, based on the representative parallax for each of thestereoscopic image frames acquired by the representative parallaxacquisition unit; a step of adjusting by the output parallax adjustmentunit an output parallax for each of the stereoscopic image frames, basedon the provisional output parallax for each of the stereoscopic imageframes determined by the provisional output parallax determination unit;a step of sequentially outputting by the output unit each of thestereoscopic image frames whose output parallax is adjusted by theoutput parallax adjustment unit; a step of determining by theprovisional output parallax determination unit the provisional outputparallax for a reference frame corresponding to the representativeparallax for the reference frame, and determining the provisional outputparallax for a target frame corresponding to the representative parallaxfor the target frame, in accordance with the provisional output parallaxdefined for each range of the representative parallax for thestereoscopic image frame, the reference frame being sequentiallydetermined from the stereoscopic image frames, and the target framebeing a stereoscopic image frame immediately after the reference frame;and a step of adjusting by the output parallax adjustment unit adifference between the output parallax for the reference frame and theoutput parallax for the target frame, based on a difference between therepresentative parallax for the reference frame and the representativeparallax for the target frame, wherein the representative parallax foreach of the stereoscopic image frames includes a statistical operationvalue to be calculated based on parallaxes that, of parallaxes for thestereoscopic image frame, meet a predetermined condition; and thestatistical operation value to be calculated based on the parallaxesthat meet the predetermined condition excludes an average parallax in apredetermined area of the stereoscopic image frame.
 18. A non-transitorycomputer-readable medium having a program causing the image outputdevice to execute the image output method according to claim 17.